Estimating the unseen: A sublinear-sample canonical estimator of distributions

We introduce a new approach to characterizing the unobserved portion of a distribution, whichprovides sublinear-sample additive estimators for a class of properties that includes entropy anddistribution support size. Together with the lower bounds proven in the companion paper [29],this settles the longstanding question of the sample complexities of these estimation problems (upto constant factors). Our algorithm estimates these properties up to an arbitrarily small additiveconstant, using O(n/ log n) samples; [29] shows that no algorithm on o(n/ log n) samples canachieve this (where n is a bound on the support size, or in the case of estimating the supportsize, 1/n is a lower bound the probability of any element of the domain). Previously, no explicitsublinear-sample algorithms for either of these problems were known.Additionally, our algorithm runs in time linear in the number of samples used. Think not, because no man sees,Such things will remain unseen.–Henry Wadsworth Longellow, from “The Builders”.